[aws_billing] Add new `aws_billing` integration #14318

gpop63 · 2025-06-25T19:50:36Z

Summary

This PR introduces a new integration for ingesting AWS Cost and Usage Report (CUR) 2.0 data from S3 buckets. The integration uses S3 polling with a minimum 24-hour interval (default) to avoid duplicate ingestion, as AWS CUR files are cumulative and updated multiple times daily.

The integration includes an hourly ES transform that groups billing data by key dimensions (service, account, usage type, region, tags) and aggregates unblended costs. This transform deduplicates the raw CUR data by creating uniquely identifiable data points.

For future upgrades, the transform configuration will require bumping both the fleet_transform_version and the dest.index. The move_on_creation: true setting is required as it ensures the destination index becomes the only member of the alias when created, automatically removing previous indices from the alias. This allows dashboards to continue displaying the data correctly.

I tested an actual upgrade by adding a new field to the transform's group by, then bumping the fleet_transform_version and dest.index. On upgrade, a new index aws_billing.billing_latest-v2 was created and got the aws_billing.billing_latest alias. The old index aws_billing.billing_latest-v1 lost the alias. The dashboards kept working fine because they use the alias, not the specific index names.

The integration uses defaults specific for the AWS CUR usecase:

bucket_list_interval is set to 24h to avoid duplicate data
content_type defaults to text/csv, the standard CUR format
file_selectors defaults to \.csv\.gz$ (this doesn't work as a default yet)

Removed Configuration Options for `aws-s3`

I removed some config options that I thought are not relevant or applicable to this integration.

Option
`expand_event_list_from_field`
`include_s3_metadata`
`max_number_of_messages`
`queue_url`
`sqs_max_receive_count`
`sqs_wait_time`
`visibility_timeout`

Minor inconveniences

Transform name starts with logs-* for some reason, this doesn't affect the functionality
~~Unable to set file_selectors default value to \.csv\.gz$~~

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

packages/aws_billing/data_stream/billing/manifest.yml

elastic-vault-github-plugin-prod · 2025-06-25T20:23:32Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/aws_billing/changelog.yml

mykola-elastic · 2025-06-30T11:26:35Z

packages/aws_billing/manifest.yml

+  - custom
+conditions:
+  kibana:
+    version: "^8.18.0"


is it not meant to be supported on ^9.0.0?

the functionality we need is already available in version 8.18.0 (probably even in older versions), so unless there are significant bug fixes in the filebeat awss3 input or other stack features in general, there's no strong justification for bumping the version 🤔

we could change it to ^8.16.5 || ^9.0.0"

@gpop63 do you want to support only 8.x, or also 9.x? If you want to support both, you need to add || ^9.0.0.

gpop63 · 2025-07-02T18:38:41Z

Dashboards updated @agithomas @daniela-elastic

rename pipeline update pipeline tests

packages/aws_billing/data_stream/cur/sample_event.json

packages/aws_billing/changelog.yml

...ng/data_stream/cur/_dev/test/pipeline/test-aws-billing-without-resource-id.log-expected.json

packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml

packages/aws_billing/data_stream/cur/fields/fields.yml

packages/aws_billing/manifest.yml

fix duplicate dashboard file

packages/aws_billing/data_stream/cur/manifest.yml

Co-authored-by: muthu-mps <[email protected]>

Copilot

Pull Request Overview

This PR introduces a new AWS Cost and Usage Report (CUR 2.0) integration that polls S3 for billing files and provides prebuilt Kibana dashboards to visualize aggregated cost data over different time ranges.

Added validation exclusions for specific checks.
Created a manifest.yml defining package metadata, input variables, and an AWS S3 policy template.
Added three Kibana dashboards: Last month, Last 3 months, and Last 6 months.

Reviewed Changes

Copilot reviewed 28 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/aws_billing/validation.yml	Excludes SVR00002 and SVR00003 checks
packages/aws_billing/manifest.yml	Defines package metadata, vars, and aws-s3 input
packages/aws_billing/kibana/dashboard/aws_billing-fb79da15-4c4c-453f-9d3d-b792ce2482ff.json	Dashboard for Last 6 months
packages/aws_billing/kibana/dashboard/aws_billing-fa467f8d-50d9-4cb0-ae88-94ef24e9941a.json	Dashboard for Last month
packages/aws_billing/kibana/dashboard/aws_billing-a716ad8e-0f77-4592-b497-6c841447efd2.json	Dashboard for Last 3 months

Comments suppressed due to low confidence (3)

packages/aws_billing/manifest.yml:105

[nitpick] The GitHub repository reference looks incorrect; update it to the correct integration repo (e.g., elastic/obs-integration-team).

  github: elastic/obs-infraobs-integrations

packages/aws_billing/kibana/dashboard/aws_billing-fb79da15-4c4c-453f-9d3d-b792ce2482ff.json:343

[nitpick] This panel has an empty title; consider adding a descriptive name to improve dashboard readability.

                "title": "",

packages/aws_billing/kibana/dashboard/aws_billing-fb79da15-4c4c-453f-9d3d-b792ce2482ff.json:502

[nitpick] The title "Table " is generic and includes a trailing space; rename it to a meaningful description and remove extra whitespace.

                        "title": "Table ",

packages/aws_billing/kibana/dashboard/aws_billing-a716ad8e-0f77-4592-b497-6c841447efd2.json

agithomas · 2025-07-16T06:20:37Z

packages/aws_billing/_dev/build/docs/README.md

+#### Required CUR Export Configuration
+
+When creating your CUR export, the following **configuration is required** for compatibility:
+


Lets mention about the need to selection the following options, explicitly

How to navigate to the Data Exports options. It is not covered anywhere.

The user must select the Standard data export option

The time granularity setting is not mentioned . (Hourly / Daily / Monthly). Can you test if all the above options are supported as part of this integration.?

How to navigate to the Data Exports options. It is not covered anywhere.

You can follow AWS documentation for setup: 🔗 [Creating Cost and Usage Reports in AWS](https://docs.aws.amazon.com/cur/latest/userguide/cur-create.html)

The user must select the Standard data export option

### AWS Cost and Usage Report (CUR) Export To use this integration, you must first create a **standard CUR 2.0 data export** in AWS. This will automatically create an S3 bucket if one does not already exist.

The time granularity setting is not mentioned . (Hourly / Daily / Monthly). Can you test if all the above options are supported as part of this integration.?

Data granularity does not matter for this integration, it's just the number of rows present in the report that will be affected.

If i choose a monthly granularity, would the file be generated on a daily basis or on a monthly basis?

Report would be generated at the same frequency (mostly daily) it's just that higher granularity will have more rows in the report.

For example, if an EC2 instance runs continuously for 24 hours:

With hourly granularity, the CUR will include 24 line items for that instance, one per hour.

With daily granularity, it will include 1 aggregated line item for the entire day.

With monthly granularity, it will have 1 aggregated line item for that same instance for the entire month (it can still change under specific conditions)

packages/aws_billing/_dev/build/docs/README.md

elastic-sonarqube · 2025-07-16T08:06:41Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
85.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml

agithomas

LGTM!

muthu-mps

@gpop63 - Make sure the integration upgrade scenario is validated with transforms.
LGTM!

gpop63 · 2025-07-17T16:43:53Z

@elastic/integrations-triaging hello 👋 can I get a review please? Code owner approval is required from this team.

jsoriano

Approving to unblock, adding some questions and suggestions.

jsoriano · 2025-07-17T16:56:26Z

packages/aws_billing/_dev/build/docs/README.md

@@ -0,0 +1,96 @@
+# AWS CUR 2.0 Billing


How does this package relate to the billing data stream in the AWS package? https://github.com/elastic/integrations/tree/main/packages/aws/data_stream/billing

that one uses the Cost and Usage API this one uses Data Export CUR 2.0 reports, which provide much more granularity. CUR 2.0 provifdes detailed billing data that enables deeper and more precise cost analysis, whereas the Cost and Usage API offers more generalized estimates

jsoriano · 2025-07-17T17:02:00Z

packages/aws_billing/manifest.yml

+  - custom
+conditions:
+  kibana:
+    version: "^8.18.0"


@gpop63 do you want to support only 8.x, or also 9.x? If you want to support both, you need to add || ^9.0.0.

jsoriano · 2025-07-17T17:10:31Z

packages/aws_billing/validation.yml

@@ -0,0 +1,4 @@
+errors:
+  exclude_checks:
+    - SVR00002


Couldn't the dashboards of this package contain filters? See https://github.com/elastic/elastic-package/blob/main/docs/howto/update_major_package_spec.md#expected-filter-in-dashboard-.

We use a transform in this integration, the documents that we display are from that transform not from the raw data stream index, that's why I didn't add a dataset filter.

elasticmachine · 2025-07-17T18:17:41Z

💔 Build Failed

Buildkite Build
Commit: a35be62

Failed CI Steps

:sonarqube: Continuous Code Inspection

History

💚 Build #28627 succeeded 2eec6f4
💚 Build #28436 succeeded 98e77bd
💚 Build #28357 succeeded 16cfd13
💚 Build #28266 succeeded ff5f023
💚 Build #28003 succeeded 2f9109c

cc @gpop63

elastic-vault-github-plugin-prod · 2025-07-18T08:42:03Z

Package aws_billing - 0.1.0 containing this change is available at https://epr.elastic.co/package/aws_billing/0.1.0/

gpop63 self-assigned this Jun 25, 2025

gpop63 added the New Integration Issue or pull request for creating a new integration package. label Jun 25, 2025

gpop63 commented Jun 25, 2025

View reviewed changes

packages/aws_billing/data_stream/billing/manifest.yml Outdated Show resolved Hide resolved

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Jun 25, 2025

gpop63 added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label Jun 25, 2025

agithomas added the Integration:aws AWS label Jun 26, 2025

andrewkroh removed the Integration:aws AWS label Jun 26, 2025

add aws_billing integration

6651b5e

gpop63 force-pushed the enhancement/202-aws-billing-cur branch from a8077d3 to 6651b5e Compare June 26, 2025 15:33

gpop63 added 2 commits June 26, 2025 18:59

fix sample event

2ebe919

update docs

2f71d1b

gpop63 marked this pull request as ready for review June 26, 2025 17:27

gpop63 requested a review from a team as a code owner June 26, 2025 17:27

check for null region code in pipeline

00bf033

mykola-elastic reviewed Jun 30, 2025

View reviewed changes

packages/aws_billing/changelog.yml Outdated Show resolved Hide resolved

mykola-elastic reviewed Jun 30, 2025

View reviewed changes

packages/aws_billing/changelog.yml Outdated Show resolved Hide resolved

mykola-elastic reviewed Jun 30, 2025

View reviewed changes

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

gpop63 added 5 commits July 2, 2025 21:41

update dashboards

7f68beb

process cost category field

42ab761

rename pipeline update pipeline tests

add integration screenshots

85dafba

update sample event

2f9109c

fix: 2 decimals for total unblended cost

ff5f023

Linu-Elias reviewed Jul 10, 2025

View reviewed changes

packages/aws_billing/data_stream/cur/sample_event.json Show resolved Hide resolved

muthu-mps reviewed Jul 10, 2025

View reviewed changes

packages/aws_billing/manifest.yml Outdated Show resolved Hide resolved

packages/aws_billing/manifest.yml Outdated Show resolved Hide resolved

gpop63 added 2 commits July 10, 2025 12:25

fix ecs version and description

0dc8068

use observability category

e9ff35d

fix duplicate dashboard file

remove changelog comment

b3c3dd4

muthu-mps reviewed Jul 10, 2025

View reviewed changes

packages/aws_billing/data_stream/cur/manifest.yml Outdated Show resolved Hide resolved

gpop63 and others added 4 commits July 10, 2025 12:40

make total and bundled discount separate fields

e4a466f

fix dashboard file name

429d4a3

Update packages/aws_billing/data_stream/cur/manifest.yml

1fdd33e

Co-authored-by: muthu-mps <[email protected]>

add fields descriptions

16cfd13

gpop63 requested a review from muthu-mps July 10, 2025 09:54

muthu-mps requested a review from Copilot July 11, 2025 10:16

Copilot AI reviewed Jul 11, 2025

View reviewed changes

packages/aws_billing/kibana/dashboard/aws_billing-a716ad8e-0f77-4592-b497-6c841447efd2.json Outdated Show resolved Hide resolved

fix filter in last 3 months dashboard

98e77bd

agithomas reviewed Jul 16, 2025

View reviewed changes

packages/aws_billing/_dev/build/docs/README.md Show resolved Hide resolved

update readme

2eec6f4

lucian-ioan reviewed Jul 17, 2025

View reviewed changes

packages/aws_billing/data_stream/cur/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

lucian-ioan mentioned this pull request Jul 17, 2025

[Azure Billing] Implement retry after logic to fix API throttling elastic/beats#45418

Open

agithomas approved these changes Jul 17, 2025

View reviewed changes

muthu-mps approved these changes Jul 17, 2025

View reviewed changes

jsoriano approved these changes Jul 17, 2025

View reviewed changes

add support for 9.0.0

a35be62

gpop63 merged commit 33cec19 into elastic:main Jul 18, 2025
7 checks passed

andrewkroh added the Integration:aws_billing AWS Cost and Usage Report (CUR 2.0) label Jul 18, 2025

		#### Required CUR Export Configuration

		When creating your CUR export, the following configuration is required for compatibility:

[aws_billing] Add new aws_billing integration #14318

[aws_billing] Add new aws_billing integration #14318

Uh oh!

Conversation

gpop63 commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Removed Configuration Options for aws-s3

Minor inconveniences

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Jun 25, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

mykola-elastic Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gpop63 commented Jul 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gpop63 Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gpop63 Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-sonarqube bot commented Jul 16, 2025

Quality Gate passed

Uh oh!

Uh oh!

agithomas left a comment

Choose a reason for hiding this comment

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

gpop63 commented Jul 17, 2025

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

[aws_billing] Add new `aws_billing` integration #14318

[aws_billing] Add new `aws_billing` integration #14318

gpop63 commented Jun 25, 2025 •

edited

Loading

Removed Configuration Options for `aws-s3`

mykola-elastic Jun 30, 2025 •

edited

Loading

gpop63 Jul 16, 2025 •

edited

Loading

gpop63 Jul 16, 2025 •

edited

Loading

elasticmachine commented Jul 17, 2025 •

edited

Loading